[refactor](storage) Unify BE and Recycler object clients - #66350
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
f30abce to
dc25d3e
Compare
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
run buildall |
|
/review |
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
Codex automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
FE UT Coverage ReportIncrement line coverage |
1fab2ee to
4ea09ca
Compare
|
/review |
|
run buildall |
There was a problem hiding this comment.
Static-only review of the full authoritative diff found six issues that should be addressed before merge (five P1, one P2).
Review-cycle status: incomplete after the three-round cap. Both normal agents returned NO_NEW_VALUABLE_FINDINGS in Round 3, but the risk-focused agent found a final FE-side scope correction that was independently verified and merged into the token round-trip comment; the review contract does not permit a fourth round. All currently known candidates are nevertheless adjudicated and included below.
Critical checkpoint conclusions:
- Data correctness: failed. Session-token credentials are dropped/staled across FE DDL and meta-service paths, and Recycler exists status mapping can turn real provider failures into false not-found results.
- Concurrency and lifecycle: delete-task ownership, executor waiting, batch clamping, and error propagation are sound; request admission during recursive deletion is not.
- Configuration and dynamic behavior: Recycler rate limiting and PUT fault injection are bypassed for the actual recursive-delete SDK requests; AWS provider precedence, refresh-capable providers, and client cache identity otherwise remain compatible.
- Compatibility and rolling behavior: the optional protobuf field is wire-compatible, but the Recycler 0/1/negative adapter contract and GCS iterator migration are broken.
- Parallel paths: BE/Recycler and S3/Azure/GCS paths were traced; the GCS path has an unconditional compile failure and Recycler differs from the preserved BE admission behavior.
- Tests and validation: no builds or tests were run, as required by the review prompt. Existing S3 accessor tests still require 1 for not-found, and there is no end-to-end token persistence/redaction/rotation coverage; the GCS compile error is statically evident.
- Observability and security: the session token lacks SK-equivalent encryption/log/display handling, and successful S3 writes now log at INFO on the hot path. This is credential-secret handling within authenticated control paths; no unsupported cross-tenant vulnerability claim is made.
- Persistence and recovery: token-bearing vault/stage records can either lose the token or retain it plaintext, so persistence round trips are not safe.
- Performance: recursive deletion can evade Recycler request controls, while per-write INFO logging adds log I/O proportional to storage QPS; page and provider batch limits themselves are sound.
User focus: review_focus.txt supplied no additional focus, so the entire PR was reviewed without narrowing scope.
FE UT Coverage ReportIncrement line coverage |
92a3812 to
3f0e01e
Compare
|
run buildall |
2 similar comments
|
run buildall |
|
run buildall |
|
/review |
There was a problem hiding this comment.
Review opinion
Request changes. The shared object-storage refactor is largely coherent, but it narrows the Azure exception boundary for multipart upload and HEAD. A standard exception from successful-response parsing can now escape the facade; on the BE upload pool that can terminate the process instead of completing the upload buffer with an error. I left one P1 inline comment.
Critical checkpoint conclusions
- Goal and proof: The PR consistently moves BE and Recycler S3/Azure behavior behind shared backends, authentication factories, admission control, and recursive-delete orchestration. The migration is mostly achieved, but the accepted Azure exception-boundary regression means the goal is not safely complete. The changed unit/mocked tests cover many facade, limiter, writer, credential, and recursive-delete paths, but no negative test proves conversion of non-
RequestFailedExceptionSDK failures. - Focus and clarity: Although large, the change is focused on removing duplicate object clients and relocating their callers/tests. The new facade/backend split is understandable, and no unrelated production change survived the full review.
- Concurrency: I traced the BE upload pool, client cache/reset locking, request admission, and Recycler executor submission, bounded backpressure, cancellation, waits, and first-error settlement. Locks remain scoped around lightweight shared state and no new lock-order/deadlock issue was found. The blocker is the uncaught Azure standard exception crossing the upload-worker boundary.
- Lifecycle and static state: Backend/client/credential shared ownership, AWS SDK/static lifetime, executor reset, Azure shared-key/SAS lifetime, and cross-library bvar/CMake ownership were checked. No circular ownership, shutdown leak, or cross-TU initialization-order defect was found.
- Configuration: No new configuration item is introduced. Existing endpoint, TLS/CA, credential-provider, limiter, and internal-bucket settings are propagated through the new factories; no additional dynamic-reload defect was found.
- Compatibility: No storage format, persisted metadata, FE-BE protocol, or externally serialized value is changed. Numeric status conventions, namespace aliases, request fields, and provider behavior were traced; no rolling-upgrade compatibility issue was found.
- Parallel paths: S3/Azure/GCS, AWS V1/V2 and credential modes, BE/Recycler, synchronous/executor deletion, and reader/writer/list/delete/HEAD paths were compared. The accepted Azure
upload_part/head_objectboundary is the only uncovered distinct regression; other concerns are already in live threads or were dismissed with code evidence. - Conditionals: Provider selection,
BUILD_AZURE/USE_AZURE, NOT_FOUND mapping, page termination, capability batching, and credential branches were checked against siblings. No additional incorrect or unexplained special condition was found. - Test coverage: The PR updates broad C++ unit/mock coverage, including the facade, rate limiting, writers, credentials, and recursive deletion. It lacks a negative test for an ordinary exception from Azure successful-response parsing; existing live review threads already cover other missing executor/provider cases. No new end-to-end regression test is added.
- Test results: No result/golden files are changed. I did not run local builds or tests under this review-only bundle. At submission time Cloud UT and FE UT pass; Clang Formatter fails; BE UT, macOS BE UT, compile, performance, and this review check are still pending.
- Observability: Provider errors, request IDs, HTTP status, TLS context, and bvars were traced before Recycler response collapse. Apart from concerns already raised in live threads, no additional production observability gap survived; the suspected Azure ListBlobs gap was dismissed because the pinned storage exception already includes provider message and request ID in
what(). - Transactions and persistence: No EditLog, transaction-state, master-failover, or persistent-storage schema change is introduced.
- Data writes, atomicity, and crashes: Multipart PUT, completion, reads, batch deletes, recursive deletion, retry/idempotency, and partial-failure settlement were traced. No new atomicity issue was found, but the accepted Azure exception path can escape the upload worker and terminate the BE instead of returning an error, so this checkpoint is blocking.
- FE-BE variables: No new variable or thrift/protobuf value needs propagation between FE and BE.
- Performance: Page and provider batch sizes, bounded streaming deletion, admissions, cache reuse, allocations, and hot-path logging were reviewed. No additional substantiated CPU, memory, complexity, or redundant-work regression was found beyond concerns already covered in live threads.
- Other issues: Authentication and cloud trust-boundary changes were checked against
SECURITY.mdand the threat model; no new distinct security issue survived deduplication. No additional nullability, ownership, memory-safety, or stale-symbol defect was found.
Review completion and focus
The focus file contains no extra user guidance, so I reviewed the entire 64-file PR. Three full convergence rounds completed; all final reviewers returned NO_NEW_VALUABLE_FINDINGS against the same frozen one-comment set. Every candidate is accepted, merged, duplicated, already covered, or dismissed with concrete evidence. Immediately before submission, the live head/base still matched 5f7c022169e34d78936527cd3a653d2bb564f93b / 07ba0d1e448be58bef46e17c01498cb3d9864891, and no live review, inline, or issue comment duplicated the accepted finding.
### What problem does this PR solve? Issue Number: None Related PR: apache#66350 Problem Summary: Azure multipart upload and HEAD response parsing could throw standard-library exceptions that escaped the object-storage facade and could terminate an upload worker. The Azure client factory aggregate also failed BE UT compilation under -Wmissing-designated-field-initializers. Catch and convert standard exceptions with path context, default-initialize all factory result fields, and merge the latest master while preserving the common object-client include in the embed tests. ### Release note None ### Check List (For Author) - Test: Not run at the author request; formatted modified C++ files with clang-format 16 and checked their diffs with git diff --check - Behavior changed: Yes, Azure SDK response parsing failures now return INTERNAL_ERROR instead of escaping - Does this need documentation: No
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
### What problem does this PR solve? Issue Number: None Related PR: apache#66350 Problem Summary: The clang-format GitHub Actions job rejected the FileInfo aggregate initialization in S3FileSystem::list_impl. Apply the clang-format 16 layout expected by the repository formatter. ### Release note None ### Check List (For Author) - Test: Ran ./build-support/check-format.sh successfully; no compilation or runtime tests were needed for a formatting-only change - Behavior changed: No - Does this need documentation: No
|
run buildall |
|
/review |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
|
run buildall |
|
/review |
|
run buildall |
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: None Related PR: apache#66350 Problem Summary: BE UT compilation treats missing designated aggregate fields as errors. AwsCredentialFactory returns AwsCredentialResult with only provider or error initialized, causing seven -Wmissing-designated-field-initializers failures. Default-initialize both result fields so partial designated initializers are valid and consistent with the Azure credential result. ### Release note None ### Check List (For Author) - Test: Not run at the author request; formatted the modified header with clang-format 16 and checked with git diff --check - Behavior changed: No - Does this need documentation: No
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#66350 Problem Summary: BE UT builds with PCH disabled, and s3_client_factory_test.cpp directly uses io::ObjClientHolder without including the header that declares it. Add the direct s3_file_system.h include so the test translation unit compiles independently. ### Release note None ### Check List (For Author) - Test: Local BE UT was not run; clang-format 16 dry-run and git diff --check passed. - Behavior changed: No - Does this need documentation: No
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
1 similar comment
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 28951 ms |
TPC-DS: Total hot run time: 166727 ms |
ClickBench: Total hot run time: 23.91 s |
1. What does this PR do?
BE and Cloud Recycler previously maintained separate object-storage abstractions and separate S3/Azure implementations. Although both sides called the same cloud-provider SDKs, credential construction, error conversion, metrics, pagination, batch deletion, and compatibility behavior were duplicated and could evolve differently.
This PR consolidates the implementation under
common/cpp/clientand exposes oneObjStorageClientfacade to upper layers:ObjStorageClientowns backend-independent orchestration and is the only complete client used by BE and Recycler call sites.ObjStorageRateLimitPolicykeeps BE- and Recycler-specific admission behavior injectable without coupling common code to either environment.ObjStorageBackendis the storage implementation boundary, implemented byS3ObjStorageBackendandAzureObjStorageBackend.Before the refactor, BE and Recycler reached the cloud SDKs through parallel stacks:
After the refactor, BE and Recycler stay on the left and right while the shared facade and backend components are centered below them:
This design keeps current BE and Recycler production call sites behind the policy-bearing facade, avoiding accidental policy bypass. Backend code implements cloud mechanics; common orchestration and policy dispatch remain in the facade.
2. How are the different behaviors unified?
doris::io::ObjStorageClientand eager list resultsdoris::ObjStorageClientfacade and one set of request/response types.doris::ioaliases keep BE call sites source-compatible; Recycler adapters preserve its integer-facing API.S3ObjStorageBackendandAzureObjStorageBackendare shared by both callers.ObjStorageRateLimitPolicy; the facade performs admission immediately before backend work. Each list page and each backend-sized delete batch is admitted independently.ObjectStorageResponsecarries a Doris status code and preserves an HTTP code and request ID when available, while adapters preserve caller-facing behavior such as the Recycler 0/1/negativeexistscontract.ObjStorageClient::list_objectsreturns one fixed-sizeObjectStorageListPage. For an admitted page request, one Client call maps to one Backend call and one SDK request. The upperObjectListIteratorowns the continuation token and requests the next page only after its cached page is consumed.END_OF_FILEis an internal upper-iterator sentinel andnext()converts it to a successful empty result. BackendNOT_FOUNDremains a real error.NoSuchKeyhandling existed in the BE pathNoSuchKey-as-empty behavior once for both callers.SyncExecutor; BE uses the synchronous fallback.AwsCredentialFactoryimplements static credentials, default provider chains, role ARN, and external ID once while callers retain their prior empty-credential behavior.AzureAuthFactorycreates the container client and shared-key credential for both. The BE factory passes TLS diagnostic context into the shared backend, whose common conversion helpers append it to matching TLS/CA failures.client_bvar::ScopedLatencytimer and common failure metrics. The previous BE 5-second slow-request logging behavior is retained for S3 uploads.3. Design boundaries and follow-ups
std::shared_ptr<ObjStorageClient>; they do not storeObjStorageBackenddirectly.ObjectListIteratorperforms lazy iteration by repeatedly calling the one-page Client API. Each page request attempts one facade admission and, once admitted, issues one SDK request; reading objects already cached in that page performs no network request.delete_objectscall may split input according to backend capability, but each resulting backend batch acquires its own PUT admission before issuing one SDK request.SyncExecutor.4. How was this PR tested?
git diff --checkpassed.